System and method for surgical guidance and intra-operative pathology through endo-microscopic tissue differentiation

ABSTRACT

Systems and methods for image classification include receiving imaging data of in-vivo or excised tissue of a patient during a surgical procedure. Local image features are extracted from the imaging data. A vocabulary histogram for the imaging data is computed based on the extracted local image features. A classification of the in-vivo or excised tissue of the patient in the imaging data is determined based on the vocabulary histogram using a trained classifier, which is trained based on a set of sample images with confirmed tissue types.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.15/568,219 filed on Oct. 20, 2017, which is a U.S. National PhaseApplication of PCT/US2015/030095 filed on May 11, 2015, the contents ofwhich are incorporated by reference.

BACKGROUND OF THE INVENTION

The present invention relates generally to surgical guidance and tissuedifferentiation, and more particularly to surgical guidance andintra-operative pathology through endo-microscopic tissuedifferentiation.

The therapy of choice for most malignant and benign tumors in the humanbody is the surgical attempt aimed at total resection of the tumor withpreservation of normal functional tissue, followed byradio-chemotherapy. An incomplete resection of a tumor with remaininginfiltrative growing cells increases the risk of recurrence withadjacent therapies, decreases the quality of life, and elevates the riskof mortality. Diagnosis of tumor and definition of tumor bordersintra-operatively is primarily based on the visualization modalities,where for example a surgeon takes a limited number of biopsy specimenswhich are later examined through histopathology performed as quickly aspossible to provide proper feedback during the surgery. Unfortunately,intraoperative fast histopathology is often not sufficientlyinformative, due to freezing artifacts, mechanical tissue destruction,and tissue architecture alteration during the sample preparation. Inaddition, sampling errors are another source of inaccuracy. Optimalsurgical therapy, which is the combination of maximal near totalresection and minimal injury of the normal tissue, is only achieved ifthe surgeon is able to identify intra-operatively the tissue cellularstructures and differentiate tumorous from normal functional tissue.

BRIEF SUMMARY OF THE INVENTION

In accordance with an embodiment, systems and methods for imageclassification include receiving imaging data of in-vivo or excisedtissue of a patient during a surgical procedure. Local image featuresare extracted from the imaging data.8 A vocabulary histogram for theimaging data is computed based on the extracted local image features. Aclassification of the in-vivo or excised tissue of the patient in theimaging data is determined based on the vocabulary histogram using atrained classifier, which is trained based on a set of sample imageswith confirmed tissue types.

In accordance with on embodiment, systems and methods for imageregistration include extracting personalized biomechanical parametersfrom a first region of tissue of a patient in an inverse problem of thebiomechanical model using pre-operative imaging data and intra-operativeimaging data. Correspondences are identified between an outer layer of asecond region of the tissue in the pre-operative imaging data and theouter layer of the second region of the tissue in the intra-operativeimaging data. A deformation of an inner layer of the second region ofthe tissue in the pre-operative imaging data is determined based on theidentified correspondences by applying the biomechanical model with thepersonalized biomechanical parameters.

In accordance with one embodiment, systems and methods for performingtumor resection on a brain of a patient include registeringpre-operative imaging data and intra-operative imaging data. Theregistered pre-operative imaging data and intra-operative imaging dataare displayed. A confocal laser endomicroscopy (CLE) probe is navigatedto a region of in-vivo or excised brain tissue including the tumor basedon the displaying the registered pre-operative imaging data andintra-operative imaging data. CLE imaging data is received from the CLEprobe at a border of the tumor. A classification of the region of thein-vivo or excised brain tissue is determined as at least one of healthytissue and tumorous tissue. The classification of the in-vivo or excisedbrain tissue is displayed for resection of the tumor. The determiningthe classification of the region of the in-vivo or excised brain tissueand the displaying the classification of the in-vivo or excised braintissue are repeated until the displaying the classification of thein-vivo or excised brain tissue shows healthy tissue with a resectedtumor bed.

These and other advantages of the invention will be apparent to those ofordinary skill in the art by reference to the following detaileddescription and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system for image guidance and classification, inaccordance with one embodiment;

FIG. 2 shows illustrative images of tissue deformation after craniotomy,in accordance with one embodiment;

FIG. 3 shows a high-level framework for image classification, inaccordance with one embodiment;

FIG. 4 illustratively shows images having low texture information, inaccordance with one embodiment;

FIG. 5 illustratively shows local image feature sampling inside a regionof interest of an intra-operative image, in accordance with oneembodiment;

FIG. 6 shows an overview of vocabulary tree training, in accordance withone embodiment;

FIG. 7 shows an exemplary display of a workstation, in accordance withone embodiment;

FIG. 8 shows a method for image guidance and classification, inaccordance with one embodiment;

FIG. 9 shows a detailed method for registering pre-operative imagingdata and intra-operative imaging data based on a personalizedbiomechanical model, in accordance with one embodiment;

FIG. 10 shows a detailed method for classifying tissue, in accordancewith one embodiment;

FIG. 11 shows a high-level workflow for a tumor resection procedure ofthe brain, in accordance with one embodiment; and

FIG. 12 shows a high-level block diagram of a computer for imageguidance and classification, in accordance with one embodiment.

DETAILED DESCRIPTION

The present invention generally relates to surgical guidance andintra-operative pathology through endo-microscopic tissuedifferentiation. Embodiments of the present invention are describedherein to give a visual understanding of methods for surgical guidanceand tissue differentiation. A digital image is often composed of digitalrepresentations of one or more objects (or shapes). The digitalrepresentation of an object is often described herein in terms ofidentifying and manipulating the objects. Such manipulations are virtualmanipulations accomplished in the memory or other circuitry/hardware ofa computer system. Accordingly, it is to be understood that embodimentsof the present invention may be performed within a computer system usingdata stored within the computer system.

Further, it should be understood that while the embodiments discussedherein may be discussed with respect to tumor resection on the brain ofa patient, the present principles are not so limited. Embodiments of thepresent invention may be employed for guidance and classification forany procedure or any subject (e.g., mechanical systems, piping systems,etc.).

FIG. 1 shows a system 100 for image guidance and classification, inaccordance with one or more embodiments. System 100 may be employed toprovide surgical guidance during a medical (e.g., surgical) procedure(or any other type of procedure), such as a craniotomy. System 100 maybe located in a hybrid operating room environment where an imageacquisition device is readily available during the course of surgery.Elements of system 100 may be co-located (e.g., within a hybridoperating room environment or facility) or remotely located (e.g., atdifferent areas of a facility or different facilities).

System 100 includes workstation 102 for assisting a user (e.g., asurgeon) during a surgical procedure. Workstation 102 includes one ormore processors 124 communicatively coupled to data storage device 122,display device 126, and input/output devices 128. Data storage device122 stores a plurality of modules representing functionality ofworkstation 102 when executed on processor 124. It should be understoodthat workstation 102 may also include additional elements, such as,e.g., a communications interface.

Workstation 102 receives pre-operative imaging data 104 of an area ofinterest 118 of a subject 120, such as, e.g., a patient. Pre-operativeimaging data 104 is acquired prior to a procedure of area of interest118. Pre-operative imaging data 104 may be of any modality orcombination of modalities, such as, e.g., computed tomography (CT),magnetic resonance imaging (MRI), single-photon emission computedtomography (SPECT), positron emission tomography (PET), etc.Pre-operative imaging data 104 includes high resolution imaging data,such as, e.g., images, video, or any other imaging data. Area ofinterest 118 may include target objects, such as tissue of a patient(e.g., tumorous tissue), as well as other critical structures. Thetissue of the patient may be in-vivo tissue or excised tissue (e.g.,biopsied tissue). In some embodiments, pre-operative imaging data 104also includes pre-operative planning information. For example,pre-operative imaging data 104 may be annotated and marked as part of aplanning step. In one example, pre-operative imaging data 104 is markedto indicate tumor margin and important anatomical structures to beavoided. Pre-operative imaging data 104 may be received by loadingpreviously stored imaging data of subject 120 from a memory or storageof a computer system.

Workstation 102 also receives intra-operative imaging data 108 fromimage acquisition device 106 of area of interest 118 of subject 120.Intra-operative imaging data 108 is acquired during an initial phase ofthe procedure to provide a complete mapping of area of interest 118.Image acquisition device 106 may be of any modality or combination ofmodalities, such as, e.g., MRI, CT, cone beam CT, etc. Image acquisitiondevice 106 may also employ one or more probes (not shown).

Workstation 102 also receives microscopic imaging data 136 from imageacquisition device 134 of area of interest 118 of subject 120.Microscopic imaging data 136 may be received intra-operatively inreal-time during a procedure. In some embodiments, image acquisitiondevice 134 may employ one or more probes 114 for imaging area ofinterest 118 of subject 120. In one embodiment, probe 114 is anendo-microscopic probe, such as, e.g., a confocal laser endomicroscopy(CLE) probe. CLE is an imaging technique which provides microscopicinformation of tissue in real-time on a cellular and subcellular level.

Probe 114 may be instrumented with tracking device 116 as part ofnavigation system 110 for tracking the position of the tip of probe 114within the intra-operative imaging coordinate system. Tracking device116 may include an optical tracking device, an electromagnetic (EM)tracking device, a mechanical tracking system, or any other suitabletracking device. Probe 114 may also include one or more imaging devices(e.g., cameras, projectors), as well as other surgical equipment ordevices, such as, e.g., insufflation devices, incision devices, or anyother device. In some embodiments, probe 114 may be tracked andmanipulated using microrobots or micro-manipulators in combination withnavigation system 110. Image acquisition device 106 is communicativelycoupled to probe 114 via connection 112, which may include an electricalconnection, an optical connection, a connection for insufflation (e.g.,conduit), or any other suitable connection.

In one embodiment, pre-operative imaging data 104 may be acquired ofarea of interest 118 at an initial (e.g., non-deformed) state whileintra-operative imaging data 108 and microscopic imaging data 136 may beacquired of area of interest 118 at a relatively deformed state. Forexample, pre-operative imaging data 104 may include imaging data of abrain of a patient acquired prior to a craniotomy while intra-operativeimaging data 108 and microscopic imaging data 136 may include imagingdata of the brain of the patient acquired after the craniotomy (i.e.,after the opening of the skull). The opening of the skull may result ina shift or a deformation of brain structures (e.g., the tumor andcritical anatomy) due to the change in pressure (relative to before theopening of the skull). Other sources of deformation include the naturalmovement of subject 120 (e.g., breathing), insufflation, displacementdue to instruments or devices, etc. These deformations may be located inthe abdominal liver, kidney, or any other location of subject 120.

Registration module 130 is configured to register or fuse pre-operativeimaging data 104 and intra-operative imaging data 108 while compensatingfor the deformation of area of interest 118. Registration module 130computes deformations and shifts of area of interest 118 using abiomechanical model, which simulates or models movement of an organ(e.g., the brain). In one embodiment, the biomechanical model includes acontinuum mechanics model for the brain where the deformation of theentire structure can be inferred from a sparse set of knowncorrespondences within pre-operative imaging data 104 andintra-operative imaging data 108.

FIG. 2 illustratively shows images 200 of brain tissue before and afterdeformation due to a craniotomy, in accordance with one or moreembodiments. Image 202 shows a pre-operative image (e.g., ofpre-operative imaging data 104) prior to a craniotomy and image 204shows an intra-operative image (e.g., of intra-operative imaging data108) after the craniotomy. Images 200 depict cortical regions (or outerlayer regions) in a top portion (shown above the dashed lines) ofregions 206 and 208 and subcortical regions (or inner layer regions) ina lower portion (shown below the dashed lines) of regions 206 and 208.

Since the characteristics of image acquisition device 106 acquiringintra-operative image 204 are known a prior, the knowledge of theimaging accuracy of detecting certain subcortical structures is alsoknown. These areas are the primary areas which will be used to estimatethe personalized biomechanical parameters. For example, the boundarydeformation and inner voxel-based deformations may be used within aninverse problem of the biomechanical model to extract homogeneous and/ornon-homogeneous tissue properties. For example, the tissue propertiesmay include tissue elasticity and the Poisson ratio. In another example,pre-operative diffusion tensor MRI may be used to estimate materialanisotropy along fibers. Region 206 represents regions whereintra-operative imaging data 108 from image acquisition device 106 hashigher imaging accuracy of the subcortical regions while region 208represents regions where image acquisition device 106 has relativelylower imaging accuracy of the subcortical regions.

Personalized (or patient-specific) biomechanical parameters areextracted from region 206 having higher imaging accuracy of thesubcortical region. In one embodiment, an inverse problem of thebiomechanical model is solved to extract the personalized biomechanicalparameters. In an inverse problem, the biomechanical model is applied todeform region 206 of pre-operative image 202 using standard (e.g.,nominal or population based) biomechanical parameters. The biomechanicalparameters may include, e.g., tissue properties such as the tissueelasticity and Poisson ratio. Other parameters may also be employed. Insome embodiments, the biomechanical parameters may be determined basedon pre-operative image 202.

The deformed region 206 in pre-operative image 202 is compared with theactual observed deformation in region 206 of intra-operative image 204using, e.g., a similarity measure. The similarity measure may beperformed using known methods. The similarity measure result is used toupdate the parameters of the biomechanical model. This process may beiteratively repeated (e.g., for a predetermined number of times, untilthe similarity measure result is maximized or more than a thresholdvalue) to generate personalized biomechanical parameters from region206.

Once personalized biomechanical parameters are extracted from region206, correspondences between the cortical layer (i.e., the outer layerof brain) of region 208 of pre-operative image 202 and intra-operativeimage 204 are identified or established. Pre-operative image 202 isdeformed using the biomechanical model based on the correspondences andpersonalized biomechanical parameters to register pre-operative image202 and intra-operative image 204. For example, the corticalcorrespondences may be established between the brain ridges (i.e.,interface between the grey matter and cerebrospinal fluid) in bothpre-operative and intraoperative images. The intra-operative image 204could include ultrasound or surface imaging where the topology of brainsurface is captured.

Based on the patient specific biomechanical model based registration, alocation of a structure (e.g., a tumor) in an inner layer (e.g.,subcortical layer) of pre-operative image 202 can be inferred from theintra-operative image 204. The underlying reason is primary due topre-operative image 202 having much richer information compared tointra-operative image 204.

In one embodiment, to further refine the accuracy of the registration, amodel of tumor growth may be employed to estimate the extent ofinfiltrating cells, which would then locally modify the tissueproperties of the biomechanical model. Tumor growth may be modeledusing, e.g., a reaction-diffusion scheme, where the reaction termcorresponds to the cellular proliferation, and the diffusion termcorresponds to the infiltration of the tumor cells. If longitudinal datais available, the parameters of the tumor growth model may be estimatedto fit the observed tumor growth. If only one time point is available,the use of PET/SPECT data may be employed to estimate the proliferationrate parameters of the model and infer the extent of the infiltratingcells. In the area of infiltrating cells, the biomechanical models aremodified accordingly, and applied for the deformation of pre-operativeimage 202.

Workstation 102 may display the registered pre-operative imaging data104 and intra-operative imaging data 108 using display 126 for guidanceduring a procedure, such as, e.g., tumor resection on the brain. Theregistered pre-operative imaging data 104 and intra-operative imagingdata 108 may be displayed in an, e.g., overlaid configuration, side byside configuration, or any other configuration. In one embodiment, thedisplay of the registered pre-operative imaging data 104 andintra-operative imaging data 108 provides guidance for navigating probe114 acquiring microscopic imaging data 136. For example, microscopicimaging data 136 may include microscopic information of tissue on acellular and subcellular level.

Classification module 132 is configured to automatically classifymicroscopic imaging data 136 for tissue differentiation during theprocedure. For example, classification module 132 may classify tissue inmicroscopic imaging data 136 according to, e.g., healthy tissue ortumorous tissue, a particular type of tumor, a particular grade oftumor, or any other classification.

In one embodiment of classification module 132, a bag of words (e.g.,features) approach is employed for image classification. In the bag ofwords approach, global image features are represented by vectors ofoccurrence counts of visual words. For example, global image featuresmay be represented in a histogram of a vocabulary dictionary determinedbased on local image features. These global image features are then usedfor classification.

FIG. 3 shows a high-level framework 300 for image classification, inaccordance with one or more embodiments. At step 302, local imagefeatures are extracted from microscopic imaging data 136 (e.g., acquirewith CLE probe) to be classified. Images from microscopic imaging data136 with low image texture information may have less value for imageclassification. As such, these images can be preliminarily excluded fromclassification. FIG. 4 illustratively shows images 400 having lowtexture information, in accordance with one or more embodiments. Images400 may be excluded from classification.

In one embodiment, images from microscopic imaging data 136 are excludedfrom classification based on an entropy value E of each image ascompared to a predetermined threshold value. Entropy E is shown inequation (1).

E=−Σ _(iϵ(0,255)) p _(i) log (p _(i))  (1)

where p_(i) is the probability of pixel values in a region of interest Rof an image. The region of interest R may be the lens area. Probabilityp_(i) may be calculated by representing pixel values in the region ofinterest R as a histogram of image intensities inside the region ofinterest R.

Local image features are then extracted from the remaining images ofmicroscopic imaging data 136. In one embodiment, scale-invariant featuretransform (SIFT) descriptors are extracted from the remaining images asthe local image features. SIFT descriptors describe an invariant localimage structure and capture local texture information. SIFT descriptorsare computed for every n_(s) pixels inside the region of interest R ofeach remaining imaging in microscopic imaging data 136, where n_(s) canbe any positive integer. Each image is represented having a width w andheight h. Other local image features may also be employed, such as,e.g., local binary pattern (LBP), histogram of oriented gradient (HOG),or any other descriptor or any combination of descriptors.

FIG. 5 illustratively shows local image feature sampling inside regionof interest R of an image of microscopic imaging data 136, in accordancewith one or more embodiments. Each white dot in image 500 represents alocation where a SIFT descriptor is computed. In one embodiment, 128dimension SIFT features are used however other dimensions may also beemployed.

At step 304, a vocabulary tree is learned for the extracted local imagefeatures. The vocabulary tree may be utilized to construct thevocabulary dictionary. The vocabulary tree defines a hierarchicalquantization using, e.g., hierarchical k-means clustering. In oneembodiment, a complete binary (i.e., k=2) search tree structure isemployed having 2^(nd) leaf nodes, where n_(d) is the predetermineddepth of the binary tree. The leaf nodes are used as the visualvocabulary words.

The vocabulary tree is trained in an offline training stage. FIG. 6shows an overview 600 of vocabulary tree training, in accordance withone or more embodiments. Vocabulary tree training uses training data 602representing a collection of training SIFT descriptors (or any otherlocal image features) derived from a training dataset. A subsample 604of N_(v) samples are randomly selected from the training SIFTdescriptors, where N_(v) may be any positive integer. A k-meansclustering algorithm (k=2) is initially applied to the selectedsubsample 604 of the training SIFT descriptors. Generally, in an initialstep, k centroids or cluster centers (e.g., 2) are defined, one for eachcluster. Then, SIFT descriptors of subsample 604 are assigned to arespective cluster having a closest centroid. This process isrecursively applied for each resulting cluster until the tree depthreaches n_(d) to train vocabulary tree 606. Each leaf node in thevocabulary tree may be associated with a label or vocabulary describingeach leaf node. The vocabulary tree encodes most prominent discriminatefeatures extracted from images, which will then be used to classifyregions of a new image (from a new patient).

In an online stage for classifying an image of microscopic imaging data136, SIFT descriptors (i.e., feature vectors) extracted from the imageare sorted in the trained vocabulary tree. Each feature vector is passeddown the trained vocabulary tree 304 level by level by comparing therespective feature vector to the two cluster centers at each level andchoosing the closest cluster center to that respective feature vector.Vocabulary histogram 306 is computed for all SIFT descriptors on eachimage by determining a number of SIFT descriptors for the respectiveimage located at each of the leaf nodes of the trained vocabulary tree304. Vocabulary histogram 306 is a histogram of the number of localimage features from an image sorted to each vocabulary term associatedwith a leaf node in the vocabulary tree. The vocabulary histogram 306represents a global image feature for an image. Each image is summarizedbased on a frequency of appearance of a certain set of featuressignified as in the feature vocabulary. The histogram of the featurevocabulary is one way to summarize the image contents as it relates tothe various important features to identify specific tumor pattern.

A classifier 308 such as, e.g., an SVM classifier is used to classifythe image based on the global image feature (e.g., vocabulary histogram306). Classifier 308 classifies tissue, e.g., as healthy tissue ortumorous tissue, as a particular grade of tumor, as one of multipledifferent tumor types, or any other classification. Classifier 308 mayinclude any suitable classifier, such as, e.g., a random forestclassifier, a K-nearest neighbor classifier, etc. Classifier 308 may betrained in an offline training step based on vocabulary histogramsdetermined for sample images in a training set having confirmed tissuetypes. In one embodiment, a marginal space deep learning (MSDL) basedframework may be employed to perform image classification. In this case,an auto-encoder convolutional neural network (CNN) may be used to trainthe marginal space learning (MSL) classifier.

In some embodiments, instead of using hands-on local image descriptors,rotationally invariant filter banks may be utilized to generate a localresponse. In this embodiment, vocabulary tree 304 is trained usingfilter bank responses.

In another embodiment of classification module 132, instead of using ahierarchical vocabulary tree to quantize local descriptors into globalimage features, classification module 132 employs a sparse coding methodas represented in equation (2).

min_(c)Σ_(i=1) ^(N) ∥x _(i) −Bc _(i)∥² +λ|c _(i)|  (2)

B is a given code book, which may be obtained using k-means cluster, avocabulary tree (e.g., as discussed above), or directly learned fromdata. x_(i) is a local feature descriptor, such as, e.g., a SIFTdescriptor. c_(i) is the vector of sparse coefficient. Parameter λ isused to control sparsity. The goal is to learn c_(i).

Histogram z is used as a global image feature for classification.Histogram z is shown in equation (3).

$\begin{matrix}{z = {\frac{1}{M}{\sum\limits_{i = 1}^{M}c_{i}}}} & (3)\end{matrix}$

where M is the number of local descriptors on each image. Histogram z isused as a global image feature for classification. Histogram z is shownin equation (3).

In one embodiment, instead of average pooling (as in a histogram), a maxpooling function is employed as in equation (4).

z _(j)=max_(i,ϵ{1,2, . . . M}) {|c _(j1) |,|c _(j2) |, . . . ,|c _(ji)|, . . . ,|c _(jM)|,}  (4)

where z₁ is the jth element of z, corresponding to jth basis of B.c_(ji) is the jth element of the sparse coefficient. c_(i) is the ithlocal descriptor on each image. Histogram z is input into a classifier(e.g., an SVM classifier) to classify the image.

Advantageously, workstation 102 aids a user during a procedure ofsubject 120. For example, workstation 102 may display the registeredintra-operative imaging data 108 and pre-operative imaging data 104using display 126 in an, e.g., overlaid configuration, a side-by-sideconfiguration, or any other configuration to provide guidance to theuser. The display of the registered intra-operative imaging data 108 andpre-operative imaging data 104 may be augmented with the location ofprobe 114. Display 126 may further display microscopic imaging data 136with the registered images (e.g., side-by-side, overlaid, etc.). In someembodiments, display 126 displays the registered pre-operative imagingdata 104 including the planning information (e.g., annotations andmarks).

In some embodiments, workstation 102 may display different types oftissue according to, e.g., tumor type or grade, healthy tissue, etc. Forexample, workstation 102 may display results of the classification usingdisplay 126. In this manner, a surgeon or other user can confirm theresults before proceeding to the next step. The results of theclassification may be displayed on display 126 having color codedoverlays over pre-operative imaging data 104, intra-operative imagingdata 108, and/or microscopic imaging data 136. For example, red may beused to indicate tumorous tissue while green may indicate healthytissue. Other forms of visual identifiers are also contemplated. In someembodiments, microscopic imaging data 136 from prior procedures may beloaded from a memory or storage of a computer system and registered anddisplayed within a current intra-operative imaging coordinate system. Infurther embodiments, workstation 102 provides reports of image sequencestaken at different anatomical locations as indicated withinpre-operative imaging data 104 and/or post-operative images.

FIG. 7 shows an exemplary display 700 of workstation 102 using display126, in accordance with one or more embodiments. Display 700 includesview 702 showing the registered pre-operative imaging data 104 andintra-operative imaging data 108 augmented with location of probe 114.Display 700 may also include view 704 of probe 114. In this manner, auser (e.g., a surgeon) not only sees the anatomical structures asdepicted by the registered images in view 702, but also can see cellularand subcellular level structures of the tissue at the tip of (e.g., CLE)probe 114 in view 704.

FIG. 8 shows a method 800 for image guidance and classification, inaccordance with one or more embodiments. At step 802, pre-operativeimaging data of tissue of a patient is received. The pre-operativeimaging data is acquired prior to a procedure. The pre-operative imagingdata may be of any modality, such as, e.g., CT, MRI, SPECT, PET, etc.The pre-operative imaging data may be images of the tissue of thepatient at an initial (i.e., non-deformed) state. In one embodiment, thepre-operative imaging data may be received by loading previously storedimaging data of subject 120 from a memory or storage of a computersystem

At step 804, intra-operative imaging data of the tissue of the patientis received. The intra-operative imaging data may be acquired at aninitial phase of the procedure. The intra-operative imaging data may beof any modality, such as, e.g., MRI, CT, cone beam CT, etc. Theintra-operative imaging data may be of images of the tissue of thepatient at a deformed state. For example, the intra-operative imagingdata may be of brain tissue of a patient after a craniotomy.

At step 806, the pre-operative imaging data and the intra-operativeimaging data are registered based on a personalized biomechanical model.In one embodiment, the biomechanical model includes a continuummechanics model for the brain. FIG. 9 shows a method 806 for registeringthe pre-operative imaging data and the intra-operative imaging databased on the personalized biomechanical model, in accordance with one ormore embodiments.

At step 902, personalized biomechanical parameters are extracted from afirst region of tissue by solving an inverse problem of thebiomechanical model. The first region may have a higher imaging accuracyin an inner layer (e.g., subcortical layer) of the tissue. Thebiomechanical model is initially applied to deform the first region oftissue in the pre-operative imaging data using standard (e.g., nominalor population based) biomechanical parameters. The biomechanicalparameters may include, e.g., elasticity and the Poisson ratio. Thedeformed first region in the pre-operative imaging data is compared tothe first region in intra-operative imaging data using a similaritymeasure. Similarity measure results are used to update the biomechanicalparameters. This process is iteratively repeated to extract personalizedbiomechanical parameters from the first region.

At step 904, correspondences between an outer layer (e.g., corticallayer) of a second region of tissue are established between thepre-operative imaging data and the intra-operative imaging data. Thesecond region of tissue having have a lower imaging accuracy in theinner layer of tissue.

At step 906, the pre-operative imaging data is deformed using thebiomechanical model based on the correspondences and the personalizedbiomechanical parameters to register the pre-operative imaging data andintra-operative imaging data. In one embodiment, a model of tumor growthmay be employed to estimate the extent of infiltrating cells. Thebiomechanical parameters may be modified in accordance with the model oftumor growth.

Returning to FIG. 8, at step 808, microscopic imaging data of the tissueof the patient is received. The microscopic imaging data may beintra-operatively acquired during a surgical procedure. The microscopicimaging data may be acquired using a CLE probe to provide microscopicinformation of tissue on a cellular and subcellular level. The CLE probemay be tracked using a tracking device and navigation system within acommon coordinate system of the registered images.

At step 810, the microscopic imaging data are classified. For example,the microscopic imaging data may be classified as, e.g., healthy ortumorous tissue, a particular grade of tumor, a particular type oftumor. FIG. 10 shows a method 810 for classifying microscopic imagingdata, in accordance with one or more embodiments.

At step 1002, images of the microscopic imaging data are excluded fromclassification based on an entropy of the images. For example,microscopic imaging data having an entropy less than a threshold valuemay be excluded from classification as having low image textureinformation.

At step 1004, local image features are extracted from the remainingmicroscopic imaging data. The local image features may include SIFTdescriptors or any other suitable local image features, such as, e.g.,LBP, HOG, or any other local image feature or combination of local imagefeatures. The local image features may be sampled at every n_(s) pixelsof a region of interest, n_(s) is any positive integer.

At step 1006, the extracted local image features are sorted in a trainedvocabulary tree. The trained vocabulary tree may be a binary vocabularytree learned using hierarchical k-means cluster (k=2). Each extractedlocal image feature (i.e., feature vector) is passed down the trainedvocabulary tree level by level by comparing the respective featurevector with two cluster centers and choosing a closest cluster center tothat respective feature vector. The trained vocabulary tree may belearned in an offline training step using training data.

At step 1008, a vocabulary histogram is computed as a global imagefeature for each image of the remaining microscopic imaging data basedon the extracted local image features sorted in the trained vocabularytree. A number of SIFT descriptors located in the leaf nodes of thetrained vocabulary tree is determined for each respective image of theremaining microscopic imaging data to compute the vocabulary histogram.The vocabulary histogram may be based on an average pooling function ora max pooling function.

At step 1010, the tissue of the patient in the remaining microscopicimaging data is classified based on the vocabulary histogram. The tissueof the patient may be classified as, e.g., healthy or tumorous, aparticular grade of tumor, a particular type of tumor, etc. A trainedclassifier, such as, e.g., SVM, random forest, K-nearest neighbor, orany other suitable classifier may be applied to classify the tissueaccording to grade of tumor, healthy tissue, etc.

Returning to FIG. 8, at step 812, the classification is displayed. Forexample, the classification may be displayed as color coded overlaysover the registered pre-operative imaging data and/or intra-operativeimaging data and/or the microscopic imaging data. The registeredpre-operative imaging data and intra-operative imaging data, as well asthe microscopic imaging data, may be displayed in a side-by-sideconfiguration, an overlaid configuration, or any other configuration. Alocation of a probe used to acquire the microscopic imaging data mayalso be displayed in the registered pre-operative imaging data andintra-operative imaging data.

FIG. 11 shows a high-level workflow 1100 for a tumor resection procedureof the brain, in accordance with one or more embodiments. In step 1102,pre-operative images of tissue of a brain of a patient are acquired. Thepre-operative images may be annotated or marked with planninginformation. The tissue may include a tumor. At step 1104,intra-operative images of the tissue of the brain of the patient areacquired after a craniotomy. The craniotomy causes a deformation in thetissue of the brain due to the change in pressure. At step 1106, thepre-operative images and intra-operative images are registered, e.g.,using a biomechanical model with personalized biomechanical parameters.At step 1108, the registered pre-operative images and intra-operativeimages are displayed using a display device. The registeredpre-operative images and intra-operative images may be displayed in aside-by-side configuration, an overlaid configuration, or any otherconfiguration.

At step 1110, a CLE probe is navigated to the tissue using guidance fromthe display device. The location of the CLE probe may be displayed withthe registered pre-operative images and intra-operative images using atracking device instrumented on the CLE probe. At step 1112, microscopicimages are acquired from the CLE probe of at a border of the tumor onthe tissue. At step 1114, a classification of the tissue is determinedin the microscopic images as at least one of tumorous tissue (or atype/grade of tumorous tissue) and healthy tissue. At step 1116, a colorrepresenting the classification of the tissue is displayed as beingoverlaid on the registered pre-operative images and intra-operativeimages. For example, a red overlay may indicate tumorous tissue while agreen overlay may represent healthy tissue. At step 1118, the tumor isresected. At step 1120, the classification is updated and displayed. Ifthe display only shows healthy tissue, the procedure ends at step 1122.However, if the display shows tumorous tissue, workflow 1100 returns tostep 1118 and the tumorous tissue is again resected.

Systems, apparatuses, and methods described herein may be implementedusing digital circuitry, or using one or more computers using well-knowncomputer processors, memory units, storage devices, computer software,and other components. Typically, a computer includes a processor forexecuting instructions and one or more memories for storing instructionsand data. A computer may also include, or be coupled to, one or moremass storage devices, such as one or more magnetic disks, internal harddisks and removable disks, magneto-optical disks, optical disks, etc.

Systems, apparatus, and methods described herein may be implementedusing computers operating in a client-server relationship. Typically, insuch a system, the client computers are located remotely from the servercomputer and interact via a network. The client-server relationship maybe defined and controlled by computer programs running on the respectiveclient and server computers.

Systems, apparatus, and methods described herein may be implementedwithin a network-based cloud computing system. In such a network-basedcloud computing system, a server or another processor that is connectedto a network communicates with one or more client computers via anetwork. A client computer may communicate with the server via a networkbrowser application residing and operating on the client computer, forexample. A client computer may store data on the server and access thedata via the network. A client computer may transmit requests for data,or requests for online services, to the server via the network. Theserver may perform requested services and provide data to the clientcomputer(s). The server may also transmit data adapted to cause a clientcomputer to perform a specified function, e.g., to perform acalculation, to display specified data on a screen, etc. For example,the server may transmit a request adapted to cause a client computer toperform one or more of the method steps described herein, including oneor more of the steps of FIGS. 8-11. Certain steps of the methodsdescribed herein, including one or more of the steps of FIGS. 8-11, maybe performed by a server or by another processor in a network-basedcloud-computing system. Certain steps of the methods described herein,including one or more of the steps of FIGS. 8-11, may be performed by aclient computer in a network-based cloud computing system. The steps ofthe methods described herein, including one or more of the steps ofFIGS. 8-11, may be performed by a server and/or by a client computer ina network-based cloud computing system, in any combination.

Systems, apparatus, and methods described herein may be implementedusing a computer program product tangibly embodied in an informationcarrier, e.g., in a non-transitory machine-readable storage device, forexecution by a programmable processor; and the method steps describedherein, including one or more of the steps of FIGS. 8-11, may beimplemented using one or more computer programs that are executable bysuch a processor. A computer program is a set of computer programinstructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

A high-level block diagram 1200 of an example computer that may be usedto implement systems, apparatus, and methods described herein isdepicted in FIG. 12. Computer 1202 includes a processor 1204 operativelycoupled to a data storage device 1212 and a memory 1210. Processor 1204controls the overall operation of computer 1202 by executing computerprogram instructions that define such operations. The computer programinstructions may be stored in data storage device 1212, or othercomputer readable medium, and loaded into memory 1210 when execution ofthe computer program instructions is desired. Thus, the method steps ofFIGS. 8-11 can be defined by the computer program instructions stored inmemory 1210 and/or data storage device 1212 and controlled by processor1204 executing the computer program instructions. For example, thecomputer program instructions can be implemented as computer executablecode programmed by one skilled in the art to perform the method steps ofFIGS. 8-11 and the modules of FIG. 1. Accordingly, by executing thecomputer program instructions, the processor 1204 executes the methodsteps of FIGS. 8-11 and modules of FIG. 1. Computer 1204 may alsoinclude one or more network interfaces 1206 for communicating with otherdevices via a network. Computer 1202 may also include one or moreinput/output devices 1208 that enable user interaction with computer1202 (e.g., display, keyboard, mouse, speakers, buttons, etc.).

Processor 1204 may include both general and special purposemicroprocessors, and may be the sole processor or one of multipleprocessors of computer 1202. Processor 1204 may include one or morecentral processing units (CPUs), for example. Processor 1204, datastorage device 1212, and/or memory 1210 may include, be supplemented by,or incorporated in, one or more application-specific integrated circuits(ASICs) and/or one or more field programmable gate arrays (FPGAs).

Data storage device 1212 and memory 1210 each include a tangiblenon-transitory computer readable storage medium. Data storage device1212, and memory 1210, may each include high-speed random access memory,such as dynamic random access memory (DRAM), static random access memory(SRAM), double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devicessuch as internal hard disks and removable disks, magneto-optical diskstorage devices, optical disk storage devices, flash memory devices,semiconductor memory devices, such as erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), compact disc read-only memory (CD-ROM), digital versatile discread-only memory (DVD-ROM) disks, or other non-volatile solid statestorage devices.

Input/output devices 1208 may include peripherals, such as a printer,scanner, display screen, etc. For example, input/output devices 1280 mayinclude a display device such as a cathode ray tube (CRT) or liquidcrystal display (LCD) monitor for displaying information to the user, akeyboard, and a pointing device such as a mouse or a trackball by whichthe user can provide input to computer 1202.

Any or all of the systems and apparatus discussed herein, includingelements of workstation 102, image acquisition device 106, andnavigation system 110 of FIG. 1, may be implemented using one or morecomputers such as computer 1202.

One skilled in the art will recognize that an implementation of anactual computer or computer system may have other structures and maycontain other components as well, and that FIG. 12 is a high levelrepresentation of some of the components of such a computer forillustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the present invention and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention. Those skilled inthe art could implement various other feature combinations withoutdeparting from the scope and spirit of the invention.

1. A method for image registration, comprising: extracting personalizedbiomechanical parameters from a first region of tissue of a patient inan inverse problem of the biomechanical model using pre-operativeimaging data and intra-operative imaging data; identifyingcorrespondences between an outer layer of a second region of the tissuein the pre-operative imaging data and the outer layer of the secondregion of the tissue in the intra-operative imaging data; anddetermining a deformation of an inner layer of the second region of thetissue in the pre-operative imaging data based on the identifiedcorrespondences by applying the biomechanical model with thepersonalized biomechanical parameters.
 2. The method as recited in claim1, wherein extracting personalized biomechanical parameters from a firstregion of tissue of a patient in an inverse problem of the biomechanicalmodel using pre-operative imaging data and intra-operative imaging datacomprises: determining an initial deformation in the first region of thetissue in the pre-operative imaging data by applying the biomechanicalmodel with standard biomechanical parameters; comparing the initialdeformation in the first region of the tissue in the pre-operativeimaging data with the first region of the tissue in the intra-operativeimaging data; and iteratively updating biomechanical parameters of thebiomechanical model based on the comparing to extract the personalizedbiomechanical parameters.
 3. The method as recited in claim 2, whereinthe standard biomechanical parameters include biomechanical parametersdetermined based on a population of patients.
 4. The method as recitedin claim 1, wherein: the first region of the tissue is a region ofhigher imaging accuracy in the inner layer of the tissue in theintra-operative imaging data, and the second region of the tissue is aregion of lower imaging accuracy in the inner layer of the tissue in theintra-operative imaging data.
 5. The method as recited in claim 1,wherein: the outer layer of the tissue includes a cortical layer of abrain of the patient, and the inner layer of the tissue include asubcortical layer of the brain of the patient.
 6. The method as recitedin claim 1, wherein extracting personalized biomechanical parametersfrom a first region of tissue of a patient in an inverse problem of thebiomechanical model using pre-operative imaging data and intra-operativeimaging data comprises: extracting personalized tissue elasticity andPoisson ratio of the patient.
 7. The method as recited in claim 1,further comprising: updating the personalized biomechanical parametersbased on a model of tumor growth for the patient.
 8. An apparatus forimage registration, comprising: means for extracting personalizedbiomechanical parameters from a first region of tissue of a patient inan inverse problem of the biomechanical model using pre-operativeimaging data and intra-operative imaging data; means for identifyingcorrespondences between an outer layer of a second region of the tissuein the pre-operative imaging data and the outer layer of the secondregion of the tissue in the intra-operative imaging data; and means fordetermining a deformation of an inner layer of the second region of thetissue in the pre-operative imaging data based on the identifiedcorrespondences by applying the biomechanical model with thepersonalized biomechanical parameters.
 9. The apparatus as recited inclaim 8, wherein the means for extracting personalized biomechanicalparameters from a first region of tissue of a patient in an inverseproblem of the biomechanical model using pre-operative imaging data andintra-operative imaging data comprises: means for determining an initialdeformation in the first region of the tissue in the pre-operativeimaging data by applying the biomechanical model with standardbiomechanical parameters; means for comparing the initial deformation inthe first region of the tissue in the pre-operative imaging data withthe first region of the tissue in the intra-operative imaging data; andmeans for iteratively updating biomechanical parameters of thebiomechanical model based on the comparing to extract the personalizedbiomechanical parameters.
 10. The apparatus as recited in claim 9,wherein the standard biomechanical parameters include biomechanicalparameters determined based on a population of patients.
 11. Theapparatus as recited in claim 9, wherein the means for extractingpersonalized biomechanical parameters from a first region of tissue of apatient in an inverse problem of the biomechanical model usingpre-operative imaging data and intra-operative imaging data comprises:means for extracting personalized tissue elasticity and Poisson ratio ofthe patient.
 12. The apparatus as recited in claim 9, furthercomprising: means for updating the personalized biomechanical parametersbased on a model of tumor growth for the patient.
 13. A non-transitorycomputer readable medium storing computer program instructions for imageregistration, the computer program instructions when executed by aprocessor cause the processor to perform operations comprising:extracting personalized biomechanical parameters from a first region oftissue of a patient in an inverse problem of the biomechanical modelusing pre-operative imaging data and intra-operative imaging data;identifying correspondences between an outer layer of a second region ofthe tissue in the pre-operative imaging data and the outer layer of thesecond region of the tissue in the intra-operative imaging data; anddetermining a deformation of an inner layer of the second region of thetissue in the pre-operative imaging data based on the identifiedcorrespondences by applying the biomechanical model with thepersonalized biomechanical parameters.
 14. The non-transitory computerreadable medium as recited in claim 13, wherein extracting personalizedbiomechanical parameters from a first region of tissue of a patient inan inverse problem of the biomechanical model using pre-operativeimaging data and intra-operative imaging data comprises: determining aninitial deformation in the first region of the tissue in thepre-operative imaging data by applying the biomechanical model withstandard biomechanical parameters; comparing the initial deformation inthe first region of the tissue in the pre-operative imaging data withthe first region of the tissue in the intra-operative imaging data; anditeratively updating biomechanical parameters of the biomechanical modelbased on the comparing to extract the personalized biomechanicalparameters.
 15. The non-transitory computer readable medium as recitedin claim 13, wherein: the first region of the tissue is a region ofhigher imaging accuracy in the inner layer of the tissue in theintra-operative imaging data, and the second region of the tissue is aregion of lower imaging accuracy in the inner layer of the tissue in theintra-operative imaging data.
 16. The non-transitory computer readablemedium as recited in claim 13, wherein: the outer layer of the tissueincludes a cortical layer of a brain of the patient, and the inner layerof the tissue include a subcortical layer of the brain of the patient.17. A method for performing tumor resection on a brain of a patient,comprising: registering pre-operative imaging data and intra-operativeimaging data; displaying the registered pre-operative imaging data andintra-operative imaging data; navigating a confocal laser endomicroscopy(CLE) probe to a region of in-vivo or excised brain tissue including thetumor based on the displaying the registered pre-operative imaging dataand intra-operative imaging data; receiving CLE imaging data from theCLE probe at a border of the tumor; determining a classification of theregion of the in-vivo or excised brain tissue as at least one of healthytissue and tumorous tissue; displaying the classification of the in-vivoor excised brain tissue for resection of the tumor; and repeating thedetermining the classification of the region of the in-vivo or excisedbrain tissue and the displaying the classification of the in-vivo orexcised brain tissue until the displaying the classification of thein-vivo or excised brain tissue shows healthy tissue with a resectedtumor bed.